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3.4 M-estimates and robust location estimates. M-estimators, as defined in Sec. 3.3, 
are sometimes called M-estimators of p type where the function h{$,x) may also be called 
p{9,x). Such estimators include maximum likelihood estimators as noted there. Another 
class of M-estimators is a class of estimators of location in M which are robust, meaning 
that they are not sensitive to contamination of the data by a few erroneous values, as will 
be seen. 

Let ip{$,x) be a jointly measurable function, of ^ in a parameter space and x 
in a sample space X, where i/j has values in a Euclidean space M.'^. A statistic := 
T„(Xi, ■ ■ ■ , Xn) will be called an M-estimator of V' type if i^C^m ^i) = 0. Such an 
estimator is not necessarily an M-estimator of p type, but it is related in that if p{9, x) has 
continuous first partial derivatives with respect to 9, then a necessary condition for T„ to be 
an M-estimator (of p type) is that it be one of ip type with ip = gradient of p. Under some 
rather special conditions, as for exponential families in Theorem 3.1.2, an M-estimator of 
t/j type where i/j is the gradient of p must also be one of p type. M-estimators of type, 
where '0 is not necessarily a gradient, and the definition need only hold approximately as 
n — > oo, will be further treated in the next two sections. 

A class of examples of M-estimators other than maximum likelihood estimators is 
provided by some location estimators in M as follows. 

For any probability distribution P on M with / x^dP < oo, fu, := / xdP is the unique 
m for which J{x — m)'^dP{x) is minimized. To see this, note that the latter integral is a 
quadratic function of m which is minimized where its derivative is 0. So, for observations 
in R, the sample mean X is an M-estimator for h{9,x) := {x — 9)'^. It is an estimator 
of the true mean |U, which gives a consistent sequence of estimators (by the strong law of 
large numbers) if and only if ^ is defined and finite. In this case, if we set a{x) := x^, 
then h is adjustable and a(-) is an adjustment function for any P with \x\dP{x) < oo, 
in other words whenever p is defined and finite. 

Suppose we have a location family in M, in other words a family of laws Pq, ^ G R, 
where dP0{x) = dPo{x — 9). If / \x\dPo{x) = -f-oo, then the sample means a.s. do not 
converge as n — * oo to a finite limit (RAP, Theorem 8.3.5). For example if Pq is the Cauchy 
distribution, dPo{x) :— dx/[K{l + x^)], then X has the same distribution as Xi. (This 
follows from the fact that the characteristic function /f^ e'^*^dPo{x) = e~l*l for all real t; 
it is easy to verify that the inverse Fourier transform of e~l*l is the Cauchy density.) 

A measure of location which is finite for any law P on M is the median, where m is 
a median of a random variable X (with law P) iff both P{X < m) > 1/2 and P{X > 
m) > 1/2. If there is only one median it is called the median. If X has no atoms in its 
distribution, in other words P{X = x) = for all a;, then m is a median of P if and 
only if P{X < m) = 1/2. A median always exists but may not be unique: for example if 
P{X = 0) = P{X = 1) = 1/2, then all x in [0, 1] are medians. Let F be the distribution 
function of X or P, F{t) = P{X < t) for all real t. Then F is always right-continuous and 
has left limits 

F{t-) := limP(s) = P(X<t) 

sit 

for all t. We can equivalently define a median of X or P as a t such that F{t—) < 1/2 < 
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F{t). Let m£ := inf{x : F(x) > 1/2} and := sup{y : F{y-) < 1/2}. For any 
y < mg, we have F{y—) < F{y) < 1/2, thus rriu > mf . It is then clear that the set of all 
medians of X or P is exactly the non-empty closed interval [mg,my]. If nii < rriu then we 
have F(t) — 1/2 for rrii <t < rriu, and, by right-continuity, also for t — nii. 

Let Pn be the empirical measure Pn := ^ Yl7=i ^Xi: where 5x is a point mass at x. 
Then a sample median is any median of P^- ^fn — 2r is even, then a sample median is any 
point in the interval between order statistics [^(r), ^(r+i)]^ which is the interval [mg,my] 
for Pn in that case. If n = 2r — 1 is odd, then the sample median is unique and equals 

If / |a;|(iP < oo, then m is a median for P if and only if it minimizes J \x — m\dP, so a 
sample median is an M-estimator for h{9,x) := \x — 9\, and the condition / \x\dP{x) < oo 
can be removed by an "adjustment," replacing h{9, x) by h{9, x) — a{x) for a{x) := |a;| in 
this case, as was mentioned in Sec. 3.3 and will now be shown. 

3.4.1 Proposition, (a) For any law P on M, m is a median of P if and only if 

f \x — 9\ — \x\dP{x) is minimized for 9 = m. (b) If / \x\dP{x) < oo, then m is a median of 

P if and only if J \x — 9\dP{x) is minimized for 9 = m. 

Proof. Clearly (b) follows from (a) by adding the finite constant / \x\dP{x) to the quantity 
being minimized, so it will suffice to prove (a). 

Let h{9,x) := \x — 9\ — \x\. For any fixed value of x, clearly h{-,x) is a convex 
function of 9. We have \h{9,x)\ < \9\ for all x and 9. For fixed 9, h{9,-) is a bounded, 
continuous function of x. Let ^{9) := £"(1^ ^ ^1 ^ 1^1) = Eh{9,X). The expectation is 
finite for each 9 and preserves the inequalities in the definition of convex function, so 7(-) 
is a convex function. Any convex real function of a real variable on an open interval (in 
this case, the whole line) has left and right derivatives at every point (RAP, 6.3.3). 

Whenever (f) < 9, 



h{9,x) — h{(t),x) — \x — 9\ — \x — (f)] 



We have \h{9,x) — h{(j),x)\ < \9 — (t)\ for all x, 9 and cf). 

Let F be the distribution function of P and 6 <9. Let 




g{4),9,x) 



9-(j) 



Then g{(f),9,x) = 1 for x < (f) and —1 for x > 9. For < x < 6* we have \g{(f),9,x)\ < 1, 
so the same holds for all x. If we fix (j) and let ^J,0 then g{(f), 9, x) converges boundedly to 
lx<0 — lx><^- Thus by dominated convergence, 7'(0+) = F{(j)) — (1 — F{4>)) = 2-F((/>) — 1. 
Or, if we fix 9 and let 4>^9 then gi^cf), 9, x) converges boundedly to ix<9 — ^x>9- Thus 
y(6/-) = F{9-) - (1 - F{9-)) = 2F{9-) - 1. 

If F{9) < 1/2, then 7' (6*+) < 0, so 7 is not minimized at 9. If F{9-) > 1/2, then 
Y{9—) > 0, so 7 is not minimized at 9. In either case, 9 is not a median of X. The 
remaining case is that F{9—) < 1/2 < F{9)^ in other words, ^ is a median of X. In that 
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case y{0—) < < ^'{9+). If the median is unique it is also the unique minimum of 7. Or 
if m£ < rriu, then by the non-decreasing property of the one-sided derivatives of 7 (RAP, 
6.3.3) it follows that Y{t) = for m£ < t < ruu and 7 is constant on [m^, m„], so it attains 
its minimum exactly on this interval. □ 

For a set of data, arranged as order statistics ^(1) < ■ ■ ■ < ^(n)5 if ?^ > 3 the sample 
median is unchanged if is replaced by any value s < ^(1), or if ^(n) is replaced by any 
t > On the other hand the sample mean X can be affected by a large amount for 

extreme values of s or t. A very large value of -^(n) > or a very negative value of , may 
be considered an "outlier" and one may prefer estimates that are relatively insensitive to 
outliers, in other words robust estimates. A more precise definition of robustness can be 
based on the notion of breakdown point, which in turn will require a definition of distance 
between probability measures. 

Definitions. Let {S, d) be a separable metric space and P, Q any two laws (probability 
measures on the Borel cr-algebra) on S. Then the Prokhorov metric do{P, Q) is defined by 

do{P,Q) := inf{£ > : P{A) < Q{A^) + e for aU Borel sets A} 

where := {y : d{x,y) < e for some x E A}. 
A real-valued function / on 5 is Lipschitz if 

II/IIl := svip{\ f{x) - f{y)\/d{x,y) : x y} < 00. 

Let ll/IU := sup{|/(x)| : x e S}. Let := ||/||z, + ||/||oo for any bounded, 

Lipschitz function /. Let 

dBLiP.Q) := sup{| j fd{P-Q)\: < !}• 

It is known that both the Prokhorov metric d^ and the "dual-bounded-Lipschitz" 
metric dsL or di :— dsh/'^ define metrics, for the same topology, on the set of all laws 
on S, and metrize weak convergence: for any sequence of laws on 5', as n — * 00, 
doiQn-iQ) ^ if and only if di{Qn,Q) ^ if and only if / fdQn j fdQ for every 
bounded continuous real- valued function f on S (RAP, Sec. 11.3). We have di < do (RAP, 
Corollary 11.6.5). If Q is a law on 5" and Xi, X2, ■ ■ ■ are i.i.d. variables with values in S and 
distribution Q (such Xi exist, for example as coordinates on a product 3°": RAP, Sec. 8.2), 
let Qn be the empirical measure which is the sum of measures 1/n at Xi for i — 1, ■ ■ ■ ,n. 
Then almost surely do{Qm Q) — > as n — > 00 (a theorem of V. S. Varadarajan, given in 
RAP, Sec. 11.4). 

Now estimators T„ will be considered as functions T„(P„) taking values in a locally 
compact non-compact separable metric space U such as M.^. 

Definition. Given a separable metric space S and a metric e for laws on S, a sequence 
{Tn} of estimators has breakdown point at P for e given by 6({T„}, P, e) := 
sup{£ > : for some compact set K <Z U, e{P,Q) < e implies Pr* {Tn{Qn)) ^ K) 
as n — > 00}, where the probability Pr is for Xi,X2,--- i.i.d. with distribution Q, and 
Pr*(C) := inf{Pr(i:») : D measurable, D D C} for any set C. 
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An estimator with low breakdown point, specifically, one with breakdown point 0, is 
not robust. An estimator as robust as possible should have a breakdown point as high as 
possible. In estimating location in M we will have U = 'R. The sample mean functional is 
X :— Tn{Qn) '■— I xdQn- It is easily seen that it has breakdown point for e either of 
the given metrics do or di. 

In general, breakdown points are not expected to be larger than 1/2. To see why, 
suppose that Q = {P + ^)/2 where P and /i are any two laws on S. Then do{P, Q) < 1/2, 
and also di{P, Q) < 1/2. Suppose {T„} had a breakdown point > 1/2 at P and at /U. Then 
Tn{Qn) should with high probability take values in one compact set for the definition of 
breakdown point at P, and another compact set for //. If T„ are to be consistent for some 
parameter, which is very different for P and then the compact set must be large to 
contain both parameters. This is possible in general, but can be ruled out in certain cases, 
for example as follows. 

A real- valued statistic T„ based on real observations Xi, • • • , X^ is called equivariant 
(for translation) iff 



for any real constant c. For example, X is equivariant for translation. 

3.4.2 Proposition. Let {T^} be a sequence of equivariant real-valued statistics. Let 
dPg{x) :— dPQ{x — 9), 6* e M, be a family of laws on M. Let e be do or di. Then if {T„} 
has breakdown point > 1/2 at Po for all 9, it cannot be a consistent sequence of estimators 
of 9 for any 9. 

Proof. If {Tn} is consistent for some 9, i.e. — > ^ as n — > oo for r„ = T„(Xi, • • • , X^) 
and Xi i.i.d. Pq, then {T„} is consistent for all 9 by equivariance. If {T„} has breakdown 
point r > 1/2 for ^ = 0, take the compact set K from the definition of breakdown point 
for some e with 1/2 < e < r. Then for some t large enough, K + t := {x + t : x E 
is disjoint from K. By equivariance, {T^} has breakdown point r also for 6* = t with the 
compact set K -h t working for £. Let Q := {Pq + Pt)/2. Then by definition of breakdown 
point, for n large enough, Pr* {T„((5n) ^ L} < 1 /2 ioi L = K and ioi L = K -\- but this 
implies that Pr* of the whole probability space is less than 1, a contradiction. □ 

While the do and di metrics are defined for laws on any metric space, there is another 
metric for the special case of the line M, called the Levy metric L, defined as follows. If 
laws P and Q on M have respective distribution functions F and G, let 



It is easy to check that L(-, ■) is indeed a metric for laws on M and that L{P, Q) < do{P, Q), 
considering half-lines as sets A in the definition of the Prokhorov metric do. Here do may 
be substantially larger than L: if P has mass 1/n at each of 2, 4, • ■ ■ , 2n, and Q has mass 
1/n at each of 1, 3, ■ ■ ■ , 2n — 1, then L{P,Q) = 1/n but do{P,Q) = 1. Such P and Q 
have mass "escaping to oo" and don't converge to any law. The Levy metric does metrize 
the same topology as that of do and di: for any laws Pn and P on R, do{PniP) — 



Tn(Xi + c,--- ,X„ + c) 



Tn{Xi,- ■ ■ ,Xn) + C 



L(P,Q) : 



inf{£ > : G{x-e)-e< F{x) < G{x + e) + e for all x}. 
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is equivalent to L{Pn, P) — > 0, which is equivalent to the distribution functions of Pn 
converging to that of P at all points where the latter is continuous (Helly-Bray property, 
e.g. RAP, 11.1.2, in light of 11.1.1 and 11.3.3). 

Before treating specific robust estimators further, a little general theory of robustness 
will be developed. Let e be a metric for laws and T a real-valued functional on the class 
of all laws which is continuous for e at a law Pq. Thus for e > small enough, 

h{e) := sup{|T(P)-T(Po)| : e{P,Po)<e} < oo. 

Let as before T„ = T„(P^) where P„ is an nth empirical measure for P. Let M(P,Tn) be 
the median, specifically, the smallest median of the distribution of T„ — T{P). Let 

h{e) := liminf„^ooSup{M(P,T„) : e(P,Po)<£}, 

h{e) := limsup„_^sup{M(P,T„) : e{P,Po)<e]. 

If h{e) = b{s), call their common value b{e). 

3.4.3 Proposition. For any e > 0, if T{Pn) —>■ T{P) in probability as n — > oo for every 
law P with e(P, Pq) < £, then h{e) > bi{e). 

Proof. We can assume T{Pq) = 0. For each P in the given neighborhood, since T{Pn) — > 
T(P) in probability, the median M{P,Tn) T{P) also. Thus h{e) > T{P) for each such 
P, so h{e) > bi{E) by definition oibi. □ 

Under a stronger, uniformity assumption there is a converse: 

3.4.4 Proposition. Assume that for a given e > 0, as n — > oo, e(P„, P) — in probability 
as n — > oo uniformly for P such that e(Po, P) < £, in other words for any S > 0, 

sup{Pr{e(Pn, P)>S}: e(P, Pq) < e} ^ 

as n — > oo. Then b{e) < bi{e+) := lim^|o bi{e + 7). 

Proof. It will be enough to show that for each fixed 7 > 0, b{e) < 61(6 + 7). By the 
triangle inequality, as n ^ 00 

sup{Pr{e(P,, Po) > £ + 7} : e(P,Po) < e} ^ 0. 

As soon as this supremum of probabilities is less than 1/2, the median M{P,Tn) satisfies 
M(P,T„) < sup{|T(P)-T(Q)| : e(Q,Po)<£ + 7} = 6i(e + 7)- □ 
In the case of the line and Levy metric, we have: 

3.4.5 Proposition. If the sample space is M and e is the Levy metric, then the hypothesis 
of Prop. 3.4.4 holds. 

Proof. The Glivenko-Cantelli theorem (RAP, Theorem 11.4.2) implies that empirical 
distribution functions almost surely converge uniformly on R to the true distribution 
function. Moreover, the proof of the Glivenko-Cantelli theorem shows that the convergence 
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for all distributions on R reduces to the case of U[0, 1] and so is uniform over all laws on R. 
Also, the Levy distance between two laws on M is less than or equal to the supremum of the 
absolute difference of their distribution functions. This implies that on M, L(P^, P) — ^ 
as n — > cxD (almost surely) at a rate which is uniform over all laws P on M. □ 

One can say that "the median is equivariant" in the sense that it is equivariant (for 
translation) whenever it is unique, and when there is an interval of more than one median, 
a translation of the data gives a translation of the interval of medians. The median is 
sometimes defined as the midpoint {mg + vriu)/'^ of the interval of medians or as the 
smallest median m^. 

There is a class of M-estimators of location having some properties like those of the 
median, including robustness, but which are more often, or always, unique. A non-constant 
function p(6', x) for x and 6 real will be called a wide-sense Huher function if p(6', x) = 
p{\x — $\) where p{x) = p{—x), p is convex, and p{x)/\x\ is bounded as |a;| oo. The 
convexity and symmetry properties imply that p attains its absolute minimum at (and 
perhaps elsewhere). Examples of wide-sense Huber functions include 

(a) p{x) := 

(b) p{x) := (c^ + x^)^/^ for any real c, and 

(c) p{x) :— x'^ for \x\ < b and p{x) := c\x\ — d for > h where 6 > and the other 
constants are chosen to make p continuously difi^erentiable, so that ch — d = iP' and 
26 = c, so d = 6^ and for \x\ > b, p{x) = h{2\x\ — h). 

Since Huber especially studied functions defined by (c) , they might be called "narrow- 
sense Huber functions." 

The functions in (b) and (c) are strictly convex in neighborhoods of 0, and the ones in 
(b) are strictly convex everywhere. Note that if p is convex, then the sum 'Y^^=i p{^i ~ ^) 
is convex in 9 for any Xj. Also, if p is strictly convex for < b, then the sum is strictly 
convex in a neighborhood of 9 for any 9 such that \Xi — 9\ < 6 for some i. For n large 
and b not too small, the set of such 9 will often include the set on which the sum takes 
its smallest values, so that the minimum (M-estimator) will be unique. We will always 
have uniqueness if p is strictly convex everywhere, as (c^ +a;^)^/^ is. Clearly, whenever an 
M-estimator (of p type) is unique for a wide-sense Huber function p, it will be equivariant, 
so the following shows that such estimators have the highest possible breakdown point: 

3.4.6 Proposition. Any M-estimator sequence in M where p is a wide-sense Huber func- 
tion has breakdown point 1/2 for the Levy metric or the Prokhorov metric at any law 
Po- 

Proof. Difference-quotients (p(6) — p{a))/{b — a) for < a < 6 are positive and non- 
decreasing as a — > -|-oo and/or b -|-oo, and they are bounded above by the assumption 
that p(a;)/|x| is bounded. Thus p is a Lipschitz function with 7 := ||p||l < 00 and as 
00, p{x)/{^\x\) 1 where 7 > 0. Letting a{x) := p(a;), and recalling p{9,x) = 
p{\x — 9\), it follows that \p{9,x) — a{x)\ < '-y\9\ for all x and 9. 

Let 'ip{x) :— p'{x). This is defined except for at most countably many values of x: 
p, being convex, has left and right derivatives everywhere (RAP, Corollary 6.3.3), with 
possibly p'{x—) < p'{x+) for a; in a countable set. To be specific, let il^{x) := p'{x—) 
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for all X. Then ip is non-decreasing. Now, our estimators of p type will also be of ip type 
except that the value may not be attained. Letting ijj{t,x) := ijj{x — t), an estimator 
Tn will be said to be of extended i/j type if Yl^=i '4^i.^i^i) ^ ^ foi" ^tU 9 < and < for 
aU e > T^. 

Let A(t,P) := fij{x - t)dP{x) = Ei{;{X - t) where X has law P. Then A(-,P) 
is nonincreasing in t. Also, A(t, P) increases if X becomes stochastically larger, in other 
words if X is replaced hy X + Y with y > 0. We have V-'(^) < as w — > — oo and ip{u) > 
as w ^ +0O. LetT*(P) := sup{t : A(t, P) > 0} and T**(P) := inf{t : A(t,P)<0}. 
Then T*{P) < T**{P). If T*(P) < T**{P) then X{t,P) = for T*{P) < t < T**{P). 

If Fq is the distribution function of Pq and F is that of P, then to make the distribution 
P stochastically as large as possible in the neighborhood L{P, Pq) < e, take F = G where 
G{x) := max(0, Fq{x — e) — e) for all x. The maximum value of G on M is 1 — £. G can 
be viewed as a probability distribution function on (— oo, oo] placing mass e at +oo; or, if 
desired, one can consider a law putting mass e at some large M and let M — > +oo. The 
result is the same for integrals of V', which approaches limits as a; — > oo or a; — > — oo. 

Let xq := mf{x : Fo{x) > e}. Then Fq^xq) = e, unless Pq has an atom at xq- Wc 
have G{x) = for x < a;o + ^ cind G{x) = Fq{x — e) — e for x > xq + s. Then for any F 
with L{Po,P) < e, 

/"OO 

X{t,F) < X{t,G) = / ij{x - t)dFo{x - e) + eij{+oo) 

J Xo+S 

nOO 

= / ipix + e -t)dFo{x) + s'ip{+oo), 

J Xo 

where il){+oo) := liuiu^oo il^{u), and functionals of laws are written as functionals of 
their distribution functions. Let := sup{T**(F) : L{F, Fq) < e} = T**{G). 

Symmetrically to G, let H be the distribution function for the stochastically smallest 
random variable with L{Fo, H) < e, so that H{x) := min(l, FQ{x + e)+e). Here H yields 
a distribution on [—00,00) with mass e at —00. Then h-{e) := inf{T*(F) : L(F, Fq) < 
e} = T**{G). 

As the set K in the definition of breakdown point consider the interval [b-{e), b'^{e)]. 
Let's see for which e this is compact. Now 6+(£) < 00 if for some t, \{t,G) < 0, which 
is true if and only if (1 — e)'ip{—oo) + e'ip{+oo) < 0. For our case, 'ip{—oo) = — V'(+oo) so 
the condition on e is £/(l — e) < 1 which is equivalent to £ < 1/2. Consideration of b-{e) 
leads to the same condition, e < 1/2. 

If L{Fo,F) < e, then L{Fo,Fn) < s with probability — 1 as n — 00. Then for any 
M-estimators T„ of p type, T*{H) = b-{e) < Tn < b+{e) = T**{G). These facts imply 
that the breakdown point of the sequence {Tn} is at least 1/2 for the Levy metric. 

Now since the Prokhorov metric is at least as large as the Levy metric, do > L, 
do{Po,P) < e implies L{Pq^P) < e which implies G{x) < F{x) < H{x) for all x. On the 
other hand, let y be a random variable with distribution function G. Such a Y exists having 
the following joint distribution with a random variable Xq having distribution Pq: Y — +00 
with probability e, specifically if Xq < xq and if Pq{xq) > 0, then Pr(y = +oo|Xo = a;o) 
has whatever value is needed to make Pr(y = +00) = s, while y < 00 for Xq > a^o- In 
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fact when Y is not +00 it is Xq + e. So Y is always within e of Xq, except with probabihty 
e. So the Ky Fan distance between the random variables Xq and Y is e (RAP, 9.2.2 and 
11.6.4), and it follows that the Prokhorov distance between Fq and G is < £ and so = e. 
So the breakdown point is also at least 1/2 for the Prokhorov metric. 

On the other hand, to show that the breakdown point is at most 1/2, so that it is 
exactly 1/2 for both metrics, suppose r > 1/2. For any M < 00, taking a distribution F 
close to Fq, wc will have for some S > 0, \{t,F) > S for all t < M, and likewise for F^ 
with high probability for n large. Then > M. Letting M ^ cxo, it follows that there 
is no compact set K as required in the definition of breakdown point, so the breakdown 
point is 1/2. □ 

In some cases one can prove the breakdown point is at least 1/2 by applying 3.4.2, 
but not in the generality of Prop. 3.4.6: consider the distribution with density 1/2 each 
on [0,1] and [2,3], with p{x) := \x\, so that the estimators are medians. Since sample 
medians don't converge in this case, consistency and Proposition 3.4.2 don't apply, even if 
we choose medians to be equivariant estimators. 

NOTES 

The facts in this section are as in Huber (1981). 
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